{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Object Detection with Street View House Numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook illustrates how to build a deep CNN using Keras’ functional API to generate multiple outputs: one to predict how many digits are present, and five for the value of each in the order they appear."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports & Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from keras.utils import to_categorical\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from keras.models import Sequential, Model\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Input, Dropout, BatchNormalization, Activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Best Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks](https://arxiv.org/abs/1312.6082), Goodfellow, et al, 2014"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- eight convolutional hidden layers, \n",
    "- one locally connected hidden layer\n",
    "- two densely connected hidden layers. \n",
    "- the first hidden layer contains maxout units with three filters per unit\n",
    "- the others contain rectifier units \n",
    "- the number of units is [48, 64, 128, 160] for the first four layers \n",
    "- 192 for all other locally connected layers\n",
    "- the fully connected layers contain 3,072 units each. \n",
    "- Each convolutional layer includes max pooling and subtractive normalization\n",
    "- The max pooling window size is 2 × 2. \n",
    "- The stride alternates between 2 and 1 at each layer, so that half of the layers don’t reduce the spatial size of the representation\n",
    "- All convolutions use zero padding on the input to preserve representation size. \n",
    "- The subtractive normalization operates on 3x3 windows and preserves representation size. \n",
    "- All convolution kernels were of size 5 × 5. \n",
    "- We trained with dropout applied to all hidden layers but not the input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best-performing architecture on the original dataset has eight convolutional layers and two final fully-connected layers. The convolutional layers are similar so that we can define a function to simplify their creation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def svhn_layer(model, filters, strides, n, input_shape=None):\n",
    "    if input_shape is not None:\n",
    "        model.add(Conv2D(filters, kernel_size=5, padding='same', name='CONV{}'.format(n), input_shape=input_shape))\n",
    "    else:\n",
    "        model.add(Conv2D(filters, kernel_size=5, padding='same', activation='relu', name='CONV{}'.format(n)))\n",
    "    model.add(BatchNormalization(name='NORM{}'.format(n)))\n",
    "    model.add(MaxPooling2D(pool_size=2, strides=strides, name='POOL{}'.format(n)))\n",
    "    model.add(Dropout(0.2, name='DROP{}'.format(n)))\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entire model combines the Sequential and functional API as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "svhn_layer(model, 48, 1, n=1, input_shape=(32,32,1))\n",
    "\n",
    "for i, kernel in enumerate([48, 64, 128, 160] + 3 * [192], 2):\n",
    "    svhn_layer(model, kernel, strides=2 if i % 2 == 0 else 1, n=i)\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(3072, name='FC1'))\n",
    "model.add(Dense(3072, name='FC2'))\n",
    "y = model.output\n",
    "\n",
    "n_digits = (Dense(units=6, activation='softmax'))(y)\n",
    "digit1 = (Dense(units=10, activation='softmax'))(y)\n",
    "digit2 = (Dense(units=11, activation='softmax'))(y)\n",
    "digit3 = (Dense(units=11, activation='softmax'))(y)\n",
    "digit4 = (Dense(units=11, activation='softmax'))(y)\n",
    "digit5 = (Dense(units=11, activation='softmax'))(y)\n",
    "\n",
    "svhn_model = Model(inputs=model.input, outputs=[n_digits, digit1, digit2, digit3, digit4, digit5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, the model produces six distinct outputs that we can evaluate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "CONV1_input (InputLayer)        (None, 32, 32, 1)    0                                            \n",
      "__________________________________________________________________________________________________\n",
      "CONV1 (Conv2D)                  (None, 32, 32, 48)   1248        CONV1_input[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "NORM1 (BatchNormalization)      (None, 32, 32, 48)   192         CONV1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "POOL1 (MaxPooling2D)            (None, 31, 31, 48)   0           NORM1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "DROP1 (Dropout)                 (None, 31, 31, 48)   0           POOL1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "CONV2 (Conv2D)                  (None, 31, 31, 48)   57648       DROP1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "NORM2 (BatchNormalization)      (None, 31, 31, 48)   192         CONV2[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "POOL2 (MaxPooling2D)            (None, 15, 15, 48)   0           NORM2[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "DROP2 (Dropout)                 (None, 15, 15, 48)   0           POOL2[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "CONV3 (Conv2D)                  (None, 15, 15, 64)   76864       DROP2[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "NORM3 (BatchNormalization)      (None, 15, 15, 64)   256         CONV3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "POOL3 (MaxPooling2D)            (None, 14, 14, 64)   0           NORM3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "DROP3 (Dropout)                 (None, 14, 14, 64)   0           POOL3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "CONV4 (Conv2D)                  (None, 14, 14, 128)  204928      DROP3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "NORM4 (BatchNormalization)      (None, 14, 14, 128)  512         CONV4[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "POOL4 (MaxPooling2D)            (None, 7, 7, 128)    0           NORM4[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "DROP4 (Dropout)                 (None, 7, 7, 128)    0           POOL4[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "CONV5 (Conv2D)                  (None, 7, 7, 160)    512160      DROP4[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "NORM5 (BatchNormalization)      (None, 7, 7, 160)    640         CONV5[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "POOL5 (MaxPooling2D)            (None, 6, 6, 160)    0           NORM5[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "DROP5 (Dropout)                 (None, 6, 6, 160)    0           POOL5[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "CONV6 (Conv2D)                  (None, 6, 6, 192)    768192      DROP5[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "NORM6 (BatchNormalization)      (None, 6, 6, 192)    768         CONV6[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "POOL6 (MaxPooling2D)            (None, 3, 3, 192)    0           NORM6[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "DROP6 (Dropout)                 (None, 3, 3, 192)    0           POOL6[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "CONV7 (Conv2D)                  (None, 3, 3, 192)    921792      DROP6[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "NORM7 (BatchNormalization)      (None, 3, 3, 192)    768         CONV7[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "POOL7 (MaxPooling2D)            (None, 2, 2, 192)    0           NORM7[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "DROP7 (Dropout)                 (None, 2, 2, 192)    0           POOL7[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "CONV8 (Conv2D)                  (None, 2, 2, 192)    921792      DROP7[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "NORM8 (BatchNormalization)      (None, 2, 2, 192)    768         CONV8[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "POOL8 (MaxPooling2D)            (None, 1, 1, 192)    0           NORM8[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "DROP8 (Dropout)                 (None, 1, 1, 192)    0           POOL8[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "flatten_1 (Flatten)             (None, 192)          0           DROP8[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "FC1 (Dense)                     (None, 3072)         592896      flatten_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "FC2 (Dense)                     (None, 3072)         9440256     FC1[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 6)            18438       FC2[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 10)           30730       FC2[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 11)           33803       FC2[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "dense_4 (Dense)                 (None, 11)           33803       FC2[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 11)           33803       FC2[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 11)           33803       FC2[0][0]                        \n",
      "==================================================================================================\n",
      "Total params: 13,686,252\n",
      "Trainable params: 13,684,204\n",
      "Non-trainable params: 2,048\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "svhn_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "svhn_model.compile(optimizer='adam',\n",
    "                   loss='categorical_crossentropy',\n",
    "                   metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "with pd.HDFStore('images/svhn/data.h5') as store:\n",
    "    X_train = store['train/data'].values.reshape(-1, 32, 32, 1)\n",
    "    y_train = store['train/labels']\n",
    "    X_test = store['test/data'].values.reshape(-1, 32, 32, 1)\n",
    "    y_test = store['test/labels']   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_digits = [to_categorical(d) for d in y_train.values.T]\n",
    "test_digits = [to_categorical(d) for d in y_test.values.T]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "svhn_path = 'models/svhn.cnn.weights.best.hdf5'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpointer = ModelCheckpoint(filepath=svhn_path, \n",
    "                               verbose=1, \n",
    "                               save_best_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/ipykernel_launcher.py:8: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 33401 samples, validate on 13068 samples\n",
      "Epoch 1/25\n",
      "33401/33401 [==============================] - 314s 9ms/step - loss: 9.7087 - dense_1_loss: 1.5593 - dense_2_loss: 3.0150 - dense_3_loss: 3.1225 - dense_4_loss: 1.5303 - dense_5_loss: 0.4141 - dense_6_loss: 0.0675 - dense_1_acc: 0.6416 - dense_2_acc: 0.2507 - dense_3_acc: 0.1959 - dense_4_acc: 0.6649 - dense_5_acc: 0.9424 - dense_6_acc: 0.9931 - val_loss: 7.5946 - val_dense_1_loss: 1.0219 - val_dense_2_loss: 3.0613 - val_dense_3_loss: 2.4888 - val_dense_4_loss: 0.8573 - val_dense_5_loss: 0.1596 - val_dense_6_loss: 0.0057 - val_dense_1_acc: 0.7795 - val_dense_2_acc: 0.1369 - val_dense_3_acc: 0.2273 - val_dense_4_acc: 0.7792 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00001: val_loss improved from inf to 7.59463, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 2/25\n",
      "33401/33401 [==============================] - 347s 10ms/step - loss: 10.0200 - dense_1_loss: 1.5287 - dense_2_loss: 2.9456 - dense_3_loss: 3.1548 - dense_4_loss: 1.8049 - dense_5_loss: 0.5168 - dense_6_loss: 0.0692 - dense_1_acc: 0.6140 - dense_2_acc: 0.2336 - dense_3_acc: 0.1955 - dense_4_acc: 0.6667 - dense_5_acc: 0.9380 - dense_6_acc: 0.9928 - val_loss: 6.2765 - val_dense_1_loss: 0.9886 - val_dense_2_loss: 2.0373 - val_dense_3_loss: 2.3389 - val_dense_4_loss: 0.8177 - val_dense_5_loss: 0.0916 - val_dense_6_loss: 0.0025 - val_dense_1_acc: 0.5766 - val_dense_2_acc: 0.2546 - val_dense_3_acc: 0.1850 - val_dense_4_acc: 0.8272 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00002: val_loss improved from 7.59463 to 6.27648, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 3/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 12.0488 - dense_1_loss: 1.9204 - dense_2_loss: 3.4604 - dense_3_loss: 3.4291 - dense_4_loss: 2.2792 - dense_5_loss: 0.7772 - dense_6_loss: 0.1824 - dense_1_acc: 0.5915 - dense_2_acc: 0.2173 - dense_3_acc: 0.1894 - dense_4_acc: 0.6448 - dense_5_acc: 0.9261 - dense_6_acc: 0.9866 - val_loss: 7.0373 - val_dense_1_loss: 1.1656 - val_dense_2_loss: 2.3608 - val_dense_3_loss: 2.4869 - val_dense_4_loss: 0.9296 - val_dense_5_loss: 0.0921 - val_dense_6_loss: 0.0023 - val_dense_1_acc: 0.7186 - val_dense_2_acc: 0.2559 - val_dense_3_acc: 0.2228 - val_dense_4_acc: 0.8205 - val_dense_5_acc: 0.9886 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00003: val_loss did not improve from 6.27648\n",
      "Epoch 4/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 10.5717 - dense_1_loss: 1.6174 - dense_2_loss: 2.9550 - dense_3_loss: 3.1532 - dense_4_loss: 2.0699 - dense_5_loss: 0.6426 - dense_6_loss: 0.1336 - dense_1_acc: 0.5945 - dense_2_acc: 0.2221 - dense_3_acc: 0.1899 - dense_4_acc: 0.6535 - dense_5_acc: 0.9386 - dense_6_acc: 0.9892 - val_loss: 6.7318 - val_dense_1_loss: 0.8219 - val_dense_2_loss: 2.3542 - val_dense_3_loss: 2.4985 - val_dense_4_loss: 0.9514 - val_dense_5_loss: 0.1035 - val_dense_6_loss: 0.0023 - val_dense_1_acc: 0.7293 - val_dense_2_acc: 0.1330 - val_dense_3_acc: 0.2228 - val_dense_4_acc: 0.8186 - val_dense_5_acc: 0.9868 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00004: val_loss did not improve from 6.27648\n",
      "Epoch 5/25\n",
      "33401/33401 [==============================] - 301s 9ms/step - loss: 10.4707 - dense_1_loss: 1.8261 - dense_2_loss: 2.9297 - dense_3_loss: 3.0503 - dense_4_loss: 2.0377 - dense_5_loss: 0.5622 - dense_6_loss: 0.0648 - dense_1_acc: 0.5889 - dense_2_acc: 0.2226 - dense_3_acc: 0.2073 - dense_4_acc: 0.6547 - dense_5_acc: 0.9358 - dense_6_acc: 0.9946 - val_loss: 5.7507 - val_dense_1_loss: 0.6633 - val_dense_2_loss: 2.0542 - val_dense_3_loss: 2.1970 - val_dense_4_loss: 0.7546 - val_dense_5_loss: 0.0791 - val_dense_6_loss: 0.0025 - val_dense_1_acc: 0.7681 - val_dense_2_acc: 0.2820 - val_dense_3_acc: 0.2464 - val_dense_4_acc: 0.8292 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00005: val_loss improved from 6.27648 to 5.75069, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 6/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 9.0226 - dense_1_loss: 1.4348 - dense_2_loss: 2.6869 - dense_3_loss: 3.0158 - dense_4_loss: 1.5244 - dense_5_loss: 0.3444 - dense_6_loss: 0.0162 - dense_1_acc: 0.5971 - dense_2_acc: 0.2392 - dense_3_acc: 0.1948 - dense_4_acc: 0.6762 - dense_5_acc: 0.9505 - dense_6_acc: 0.9982 - val_loss: 7.5383 - val_dense_1_loss: 1.0520 - val_dense_2_loss: 2.1542 - val_dense_3_loss: 2.9518 - val_dense_4_loss: 1.2616 - val_dense_5_loss: 0.1162 - val_dense_6_loss: 0.0025 - val_dense_1_acc: 0.6340 - val_dense_2_acc: 0.2808 - val_dense_3_acc: 0.1517 - val_dense_4_acc: 0.8238 - val_dense_5_acc: 0.9878 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00006: val_loss did not improve from 5.75069\n",
      "Epoch 7/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 9.2151 - dense_1_loss: 1.3050 - dense_2_loss: 2.6196 - dense_3_loss: 3.0590 - dense_4_loss: 1.6750 - dense_5_loss: 0.4976 - dense_6_loss: 0.0590 - dense_1_acc: 0.6211 - dense_2_acc: 0.2426 - dense_3_acc: 0.1979 - dense_4_acc: 0.6724 - dense_5_acc: 0.9399 - dense_6_acc: 0.9945 - val_loss: 8.7602 - val_dense_1_loss: 1.4513 - val_dense_2_loss: 2.5046 - val_dense_3_loss: 3.3595 - val_dense_4_loss: 1.2598 - val_dense_5_loss: 0.1825 - val_dense_6_loss: 0.0025 - val_dense_1_acc: 0.6394 - val_dense_2_acc: 0.2049 - val_dense_3_acc: 0.2309 - val_dense_4_acc: 0.8176 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00007: val_loss did not improve from 5.75069\n",
      "Epoch 8/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 9.2256 - dense_1_loss: 1.3348 - dense_2_loss: 2.7656 - dense_3_loss: 2.9522 - dense_4_loss: 1.6488 - dense_5_loss: 0.4458 - dense_6_loss: 0.0785 - dense_1_acc: 0.6076 - dense_2_acc: 0.2368 - dense_3_acc: 0.2079 - dense_4_acc: 0.6690 - dense_5_acc: 0.9459 - dense_6_acc: 0.9946 - val_loss: 5.7633 - val_dense_1_loss: 0.6405 - val_dense_2_loss: 2.0585 - val_dense_3_loss: 2.2410 - val_dense_4_loss: 0.7348 - val_dense_5_loss: 0.0862 - val_dense_6_loss: 0.0023 - val_dense_1_acc: 0.7633 - val_dense_2_acc: 0.2815 - val_dense_3_acc: 0.2723 - val_dense_4_acc: 0.8295 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00008: val_loss did not improve from 5.75069\n",
      "Epoch 9/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 8.3413 - dense_1_loss: 1.1721 - dense_2_loss: 2.5734 - dense_3_loss: 2.8296 - dense_4_loss: 1.4378 - dense_5_loss: 0.3183 - dense_6_loss: 0.0101 - dense_1_acc: 0.6214 - dense_2_acc: 0.2454 - dense_3_acc: 0.2163 - dense_4_acc: 0.6824 - dense_5_acc: 0.9518 - dense_6_acc: 0.9987 - val_loss: 6.1459 - val_dense_1_loss: 0.6629 - val_dense_2_loss: 2.2397 - val_dense_3_loss: 2.3905 - val_dense_4_loss: 0.7725 - val_dense_5_loss: 0.0778 - val_dense_6_loss: 0.0025 - val_dense_1_acc: 0.7604 - val_dense_2_acc: 0.2866 - val_dense_3_acc: 0.2414 - val_dense_4_acc: 0.8294 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00009: val_loss did not improve from 5.75069\n",
      "Epoch 10/25\n",
      "33401/33401 [==============================] - 301s 9ms/step - loss: 7.9722 - dense_1_loss: 0.9641 - dense_2_loss: 2.5649 - dense_3_loss: 2.7847 - dense_4_loss: 1.3546 - dense_5_loss: 0.2976 - dense_6_loss: 0.0062 - dense_1_acc: 0.6604 - dense_2_acc: 0.2463 - dense_3_acc: 0.2215 - dense_4_acc: 0.6827 - dense_5_acc: 0.9528 - dense_6_acc: 0.9993 - val_loss: 5.9636 - val_dense_1_loss: 0.5994 - val_dense_2_loss: 2.2489 - val_dense_3_loss: 2.2672 - val_dense_4_loss: 0.7618 - val_dense_5_loss: 0.0838 - val_dense_6_loss: 0.0024 - val_dense_1_acc: 0.7866 - val_dense_2_acc: 0.2130 - val_dense_3_acc: 0.2549 - val_dense_4_acc: 0.8293 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00010: val_loss did not improve from 5.75069\n",
      "Epoch 11/25\n",
      "33401/33401 [==============================] - 301s 9ms/step - loss: 7.9460 - dense_1_loss: 1.0460 - dense_2_loss: 2.5665 - dense_3_loss: 2.7105 - dense_4_loss: 1.2992 - dense_5_loss: 0.3177 - dense_6_loss: 0.0061 - dense_1_acc: 0.6582 - dense_2_acc: 0.2477 - dense_3_acc: 0.2201 - dense_4_acc: 0.6821 - dense_5_acc: 0.9508 - dense_6_acc: 0.9996 - val_loss: 5.9762 - val_dense_1_loss: 0.6113 - val_dense_2_loss: 2.2477 - val_dense_3_loss: 2.3819 - val_dense_4_loss: 0.6613 - val_dense_5_loss: 0.0718 - val_dense_6_loss: 0.0022 - val_dense_1_acc: 0.7749 - val_dense_2_acc: 0.2903 - val_dense_3_acc: 0.2882 - val_dense_4_acc: 0.8295 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 00011: val_loss did not improve from 5.75069\n",
      "Epoch 12/25\n",
      "33401/33401 [==============================] - 299s 9ms/step - loss: 8.7124 - dense_1_loss: 1.3090 - dense_2_loss: 2.6156 - dense_3_loss: 2.8611 - dense_4_loss: 1.5322 - dense_5_loss: 0.3541 - dense_6_loss: 0.0404 - dense_1_acc: 0.6048 - dense_2_acc: 0.2486 - dense_3_acc: 0.1979 - dense_4_acc: 0.6781 - dense_5_acc: 0.9508 - dense_6_acc: 0.9959 - val_loss: 5.8453 - val_dense_1_loss: 0.6830 - val_dense_2_loss: 2.1058 - val_dense_3_loss: 2.2155 - val_dense_4_loss: 0.7460 - val_dense_5_loss: 0.0926 - val_dense_6_loss: 0.0025 - val_dense_1_acc: 0.7609 - val_dense_2_acc: 0.2356 - val_dense_3_acc: 0.2804 - val_dense_4_acc: 0.8299 - val_dense_5_acc: 0.9886 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00012: val_loss did not improve from 5.75069\n",
      "Epoch 13/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 7.7992 - dense_1_loss: 1.1011 - dense_2_loss: 2.4061 - dense_3_loss: 2.6669 - dense_4_loss: 1.3306 - dense_5_loss: 0.2881 - dense_6_loss: 0.0064 - dense_1_acc: 0.6675 - dense_2_acc: 0.2619 - dense_3_acc: 0.2235 - dense_4_acc: 0.6909 - dense_5_acc: 0.9555 - dense_6_acc: 0.9993 - val_loss: 5.5272 - val_dense_1_loss: 0.5294 - val_dense_2_loss: 2.1043 - val_dense_3_loss: 2.1249 - val_dense_4_loss: 0.6635 - val_dense_5_loss: 0.0920 - val_dense_6_loss: 0.0131 - val_dense_1_acc: 0.8047 - val_dense_2_acc: 0.2677 - val_dense_3_acc: 0.2661 - val_dense_4_acc: 0.8315 - val_dense_5_acc: 0.9886 - val_dense_6_acc: 0.9989\n",
      "\n",
      "Epoch 00013: val_loss improved from 5.75069 to 5.52721, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 14/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 8.3126 - dense_1_loss: 1.1739 - dense_2_loss: 2.5540 - dense_3_loss: 2.8229 - dense_4_loss: 1.4067 - dense_5_loss: 0.3355 - dense_6_loss: 0.0196 - dense_1_acc: 0.6641 - dense_2_acc: 0.2491 - dense_3_acc: 0.2169 - dense_4_acc: 0.6898 - dense_5_acc: 0.9499 - dense_6_acc: 0.9984 - val_loss: 6.0660 - val_dense_1_loss: 0.7099 - val_dense_2_loss: 2.1479 - val_dense_3_loss: 2.3164 - val_dense_4_loss: 0.8092 - val_dense_5_loss: 0.0800 - val_dense_6_loss: 0.0025 - val_dense_1_acc: 0.7799 - val_dense_2_acc: 0.2821 - val_dense_3_acc: 0.2812 - val_dense_4_acc: 0.8214 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00014: val_loss did not improve from 5.52721\n",
      "Epoch 15/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 7.5857 - dense_1_loss: 0.9664 - dense_2_loss: 2.4766 - dense_3_loss: 2.6251 - dense_4_loss: 1.2467 - dense_5_loss: 0.2664 - dense_6_loss: 0.0046 - dense_1_acc: 0.7038 - dense_2_acc: 0.2580 - dense_3_acc: 0.2236 - dense_4_acc: 0.6914 - dense_5_acc: 0.9562 - dense_6_acc: 0.9995 - val_loss: 5.7650 - val_dense_1_loss: 0.5534 - val_dense_2_loss: 2.2184 - val_dense_3_loss: 2.2215 - val_dense_4_loss: 0.6872 - val_dense_5_loss: 0.0821 - val_dense_6_loss: 0.0024 - val_dense_1_acc: 0.8177 - val_dense_2_acc: 0.2360 - val_dense_3_acc: 0.2803 - val_dense_4_acc: 0.8326 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00015: val_loss did not improve from 5.52721\n",
      "Epoch 16/25\n",
      "33401/33401 [==============================] - 299s 9ms/step - loss: 7.5178 - dense_1_loss: 0.9746 - dense_2_loss: 2.3657 - dense_3_loss: 2.5889 - dense_4_loss: 1.2901 - dense_5_loss: 0.2931 - dense_6_loss: 0.0054 - dense_1_acc: 0.6835 - dense_2_acc: 0.2644 - dense_3_acc: 0.2268 - dense_4_acc: 0.6926 - dense_5_acc: 0.9549 - dense_6_acc: 0.9996 - val_loss: 5.6495 - val_dense_1_loss: 0.5974 - val_dense_2_loss: 2.0846 - val_dense_3_loss: 2.1424 - val_dense_4_loss: 0.7194 - val_dense_5_loss: 0.1038 - val_dense_6_loss: 0.0018 - val_dense_1_acc: 0.8003 - val_dense_2_acc: 0.2565 - val_dense_3_acc: 0.2804 - val_dense_4_acc: 0.8326 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00016: val_loss did not improve from 5.52721\n",
      "Epoch 17/25\n",
      "33401/33401 [==============================] - 299s 9ms/step - loss: 6.8725 - dense_1_loss: 0.8248 - dense_2_loss: 2.2207 - dense_3_loss: 2.3957 - dense_4_loss: 1.1621 - dense_5_loss: 0.2571 - dense_6_loss: 0.0121 - dense_1_acc: 0.7181 - dense_2_acc: 0.2645 - dense_3_acc: 0.2383 - dense_4_acc: 0.6961 - dense_5_acc: 0.9560 - dense_6_acc: 0.9991 - val_loss: 5.3294 - val_dense_1_loss: 0.5088 - val_dense_2_loss: 1.9996 - val_dense_3_loss: 2.0526 - val_dense_4_loss: 0.6806 - val_dense_5_loss: 0.0860 - val_dense_6_loss: 0.0017 - val_dense_1_acc: 0.8134 - val_dense_2_acc: 0.3084 - val_dense_3_acc: 0.2937 - val_dense_4_acc: 0.8285 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00017: val_loss improved from 5.52721 to 5.32939, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 18/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 5.9644 - dense_1_loss: 0.6070 - dense_2_loss: 1.9914 - dense_3_loss: 2.0994 - dense_4_loss: 1.0266 - dense_5_loss: 0.2358 - dense_6_loss: 0.0041 - dense_1_acc: 0.7590 - dense_2_acc: 0.2706 - dense_3_acc: 0.2524 - dense_4_acc: 0.6994 - dense_5_acc: 0.9568 - dense_6_acc: 0.9996 - val_loss: 5.2905 - val_dense_1_loss: 0.4665 - val_dense_2_loss: 2.0324 - val_dense_3_loss: 2.0407 - val_dense_4_loss: 0.6526 - val_dense_5_loss: 0.0965 - val_dense_6_loss: 0.0017 - val_dense_1_acc: 0.8421 - val_dense_2_acc: 0.2877 - val_dense_3_acc: 0.3045 - val_dense_4_acc: 0.8336 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00018: val_loss improved from 5.32939 to 5.29050, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 19/25\n",
      "33401/33401 [==============================] - 301s 9ms/step - loss: 5.8362 - dense_1_loss: 0.5555 - dense_2_loss: 1.9841 - dense_3_loss: 2.0715 - dense_4_loss: 0.9914 - dense_5_loss: 0.2303 - dense_6_loss: 0.0034 - dense_1_acc: 0.7807 - dense_2_acc: 0.2703 - dense_3_acc: 0.2594 - dense_4_acc: 0.7007 - dense_5_acc: 0.9566 - dense_6_acc: 0.9997 - val_loss: 5.1838 - val_dense_1_loss: 0.4323 - val_dense_2_loss: 2.0267 - val_dense_3_loss: 2.0451 - val_dense_4_loss: 0.6057 - val_dense_5_loss: 0.0721 - val_dense_6_loss: 0.0019 - val_dense_1_acc: 0.8537 - val_dense_2_acc: 0.2837 - val_dense_3_acc: 0.2984 - val_dense_4_acc: 0.8339 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00019: val_loss improved from 5.29050 to 5.18384, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 20/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 5.7492 - dense_1_loss: 0.5183 - dense_2_loss: 1.9807 - dense_3_loss: 2.0561 - dense_4_loss: 0.9647 - dense_5_loss: 0.2260 - dense_6_loss: 0.0035 - dense_1_acc: 0.8006 - dense_2_acc: 0.2722 - dense_3_acc: 0.2640 - dense_4_acc: 0.7038 - dense_5_acc: 0.9567 - dense_6_acc: 0.9997 - val_loss: 5.0958 - val_dense_1_loss: 0.4138 - val_dense_2_loss: 1.9990 - val_dense_3_loss: 2.0242 - val_dense_4_loss: 0.5869 - val_dense_5_loss: 0.0702 - val_dense_6_loss: 0.0016 - val_dense_1_acc: 0.8528 - val_dense_2_acc: 0.2821 - val_dense_3_acc: 0.2938 - val_dense_4_acc: 0.8320 - val_dense_5_acc: 0.9886 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00020: val_loss improved from 5.18384 to 5.09576, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 21/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 5.6786 - dense_1_loss: 0.4902 - dense_2_loss: 1.9743 - dense_3_loss: 2.0372 - dense_4_loss: 0.9448 - dense_5_loss: 0.2281 - dense_6_loss: 0.0039 - dense_1_acc: 0.8117 - dense_2_acc: 0.2783 - dense_3_acc: 0.2739 - dense_4_acc: 0.7070 - dense_5_acc: 0.9564 - dense_6_acc: 0.9997 - val_loss: 5.0250 - val_dense_1_loss: 0.3896 - val_dense_2_loss: 1.9856 - val_dense_3_loss: 1.9781 - val_dense_4_loss: 0.5987 - val_dense_5_loss: 0.0715 - val_dense_6_loss: 0.0014 - val_dense_1_acc: 0.8681 - val_dense_2_acc: 0.3003 - val_dense_3_acc: 0.3285 - val_dense_4_acc: 0.8332 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00021: val_loss improved from 5.09576 to 5.02496, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 22/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 5.4920 - dense_1_loss: 0.4311 - dense_2_loss: 1.9516 - dense_3_loss: 1.9804 - dense_4_loss: 0.9063 - dense_5_loss: 0.2195 - dense_6_loss: 0.0031 - dense_1_acc: 0.8393 - dense_2_acc: 0.2869 - dense_3_acc: 0.2946 - dense_4_acc: 0.7124 - dense_5_acc: 0.9567 - dense_6_acc: 0.9997 - val_loss: 5.0159 - val_dense_1_loss: 0.4236 - val_dense_2_loss: 1.9530 - val_dense_3_loss: 1.8985 - val_dense_4_loss: 0.6544 - val_dense_5_loss: 0.0848 - val_dense_6_loss: 0.0016 - val_dense_1_acc: 0.8393 - val_dense_2_acc: 0.2959 - val_dense_3_acc: 0.3415 - val_dense_4_acc: 0.8248 - val_dense_5_acc: 0.9880 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00022: val_loss improved from 5.02496 to 5.01586, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 23/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 5.3284 - dense_1_loss: 0.3936 - dense_2_loss: 1.9325 - dense_3_loss: 1.9058 - dense_4_loss: 0.8792 - dense_5_loss: 0.2143 - dense_6_loss: 0.0030 - dense_1_acc: 0.8545 - dense_2_acc: 0.3042 - dense_3_acc: 0.3127 - dense_4_acc: 0.7168 - dense_5_acc: 0.9565 - dense_6_acc: 0.9997 - val_loss: 4.4285 - val_dense_1_loss: 0.2693 - val_dense_2_loss: 1.8365 - val_dense_3_loss: 1.7403 - val_dense_4_loss: 0.5202 - val_dense_5_loss: 0.0603 - val_dense_6_loss: 0.0019 - val_dense_1_acc: 0.9082 - val_dense_2_acc: 0.3590 - val_dense_3_acc: 0.3763 - val_dense_4_acc: 0.8363 - val_dense_5_acc: 0.9887 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00023: val_loss improved from 5.01586 to 4.42847, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 24/25\n",
      "33401/33401 [==============================] - 301s 9ms/step - loss: 4.9983 - dense_1_loss: 0.3741 - dense_2_loss: 1.7137 - dense_3_loss: 1.8308 - dense_4_loss: 0.8674 - dense_5_loss: 0.2082 - dense_6_loss: 0.0041 - dense_1_acc: 0.8620 - dense_2_acc: 0.3933 - dense_3_acc: 0.3331 - dense_4_acc: 0.7158 - dense_5_acc: 0.9563 - dense_6_acc: 0.9996 - val_loss: 4.0349 - val_dense_1_loss: 0.2573 - val_dense_2_loss: 1.5541 - val_dense_3_loss: 1.6254 - val_dense_4_loss: 0.5037 - val_dense_5_loss: 0.0916 - val_dense_6_loss: 0.0029 - val_dense_1_acc: 0.9212 - val_dense_2_acc: 0.4495 - val_dense_3_acc: 0.4117 - val_dense_4_acc: 0.8397 - val_dense_5_acc: 0.9868 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00024: val_loss improved from 4.42847 to 4.03493, saving model to models/svhn.cnn.weights.best.hdf5\n",
      "Epoch 25/25\n",
      "33401/33401 [==============================] - 300s 9ms/step - loss: 4.5203 - dense_1_loss: 0.3145 - dense_2_loss: 1.5287 - dense_3_loss: 1.6466 - dense_4_loss: 0.8343 - dense_5_loss: 0.1934 - dense_6_loss: 0.0028 - dense_1_acc: 0.8868 - dense_2_acc: 0.4535 - dense_3_acc: 0.3923 - dense_4_acc: 0.7207 - dense_5_acc: 0.9566 - dense_6_acc: 0.9997 - val_loss: 3.5610 - val_dense_1_loss: 0.2380 - val_dense_2_loss: 1.3407 - val_dense_3_loss: 1.4230 - val_dense_4_loss: 0.5082 - val_dense_5_loss: 0.0494 - val_dense_6_loss: 0.0018 - val_dense_1_acc: 0.9206 - val_dense_2_acc: 0.5243 - val_dense_3_acc: 0.4994 - val_dense_4_acc: 0.8391 - val_dense_5_acc: 0.9888 - val_dense_6_acc: 0.9998\n",
      "\n",
      "Epoch 00025: val_loss improved from 4.03493 to 3.56102, saving model to models/svhn.cnn.weights.best.hdf5\n"
     ]
    }
   ],
   "source": [
    "epochs = 25\n",
    "result = svhn_model.fit(x=X_train,\n",
    "                        y=train_digits,\n",
    "                        batch_size=32,\n",
    "                        epochs=epochs,\n",
    "                        verbose=1,\n",
    "                        validation_data=(X_test, test_digits),\n",
    "                        callbacks=[checkpointer])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13068/13068 [==============================] - 33s 3ms/step\n"
     ]
    }
   ],
   "source": [
    "n_digits, digit1, digit2, digit3, digit4, digit5 = svhn_model.predict(X_test, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9206458524640343"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(y_test[0] == np.argmax(n_digits, axis=1)).sum()/len(n_digits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd8FGX+wPHPbMumk16BUFIggdCL0hQBsaBSBOvpeSqeYv/pKXrHKZyndxYsp6J4KuIBYoMTy0kRRGpoARIIJEAC6b2w2fb8/pglBEjZwIYky/N+vfa1szOzzzw7u/vdZ5955juKEAJJkiTJvWjaugKSJEmS68ngLkmS5IZkcJckSXJDMrhLkiS5IRncJUmS3JAM7pIkSW5IBndJkiQ3JIO7JEmSG5LBXZIkyQ3p2mrDwcHBIiYmpq02L0mS1CGlpKQUCSFCmluvzYJ7TEwM27dvb6vNS5IkdUiKohx1Zj3ZLSNJkuSGZHCXJElyQzK4S5IkuSEZ3CVJktyQDO6SJEluqNngrijKR4qiFCiKsreR5YqiKG8qinJIUZQ9iqIMcH01JUmSpJZwpuX+MXB1E8snArGO233AuxdeLUmSJOlCNDvOXQixXlGUmCZWuQH4VKjX69usKEonRVEihBC5LqrjGXLS93F0z87WKFrqcJTTU0pD89RpRaNBo9WicdwrjmlFo1Xnn7VMUTQIuw27zY7dZsVus2G32xA2OzabFWGzOebZsVut6jJ5ucoOwPF5UBqap5w9yzn13vYzPwOi3vxz5/UYMITwnnEt2FDLueIkpiggu97jHMe8c4K7oij3obbu6dKly3ltLPdgOpu/Wnpez5XcSHsLpkpLIoJ00bWXz4vjc+ITENghgntDn+oG96QQYgGwAGDQoEHntbcHT5rC4ElTzuepkps7o+XkmBYIhF04Wt6O1rhdbXkLu72uVW63OZbb7Qi7/XRrX6tFo9Gi0TnutVo0Wh0arcbxD0Cdp8jg3uHUfV7qfW5Ew6GrSUr9EKic+8+xwe1ehN8aVwT3HKBzvcfRwAkXlNug3auz2fzt4QsvqP6bcMb805NanQadXqPeG9R7rd4xT689vdwxT9Eq2Cx2bBY71jPubVjrzVPn27BZ7NgFaDQKGo2ColVOT2tQA4jjsUarOKbVz6LdJhBCqPd29d5uPz19atmpeW7LifexpWFXNPLgzPluvE8vVGPvSYMzcH4fKwoaBRSNUner+65oFBTl3PnCrgZT9QdevRd2Tk+LU/Op+56c/o5r0Om1dd99nUF9fGq5Tq9Ba9CCEOr32+z4npsd02b1e37mvR2b2cboW+NJHBl1ATu5ea4I7iuAhxRFWQIMBcpbq78dIDjah6TR0RdWyBm/1PXn119FDY5nBmQ7NqsNS62Nk1WWM4K4zWrHZhPoGvwh0GDw1OHp6/iA6E5/cNCoHyrhCMT2etOnPpB1Adwxran/4dYqdY/PmK73Y6EouGe3gRPv43m3kJSGp8/8AXHDfXqhGntPzp4haPE+PhWkTwVi+6nHtlPTju/Sqe+RUJ+uOAK/RoP6naj3I6Bo6v1gaBUQYLOeDsL1G2mmKgtWS23dY6vFhs1sBwV0Bq0a8A3auh8BnUGD0UfvmH96ns6gJbiz7wXv6uY0G9wVRfkPMAYIVhQlB/gLoAcQQrwHrAKuAQ4BNcDdrVVZgKj4AKLiA1pzE5IkSR2eM6NlbmlmuQAedFmNJEmSpAsmz1CVJElyQzK4S5IkuSEZ3CVJktyQDO6SJEluSAZ3SZIkNySDuyRJkhuSwV2SJMkNyeAuSZLkhmRwlyRJckMyuEuSJLkhGdwlSZLckCuyQkqSJF1STBYbu7LLSDlaSlFVLRpHOmKNRs06qT37sSM7qzpf4fKewfSO9GvVOsrgLkmS1IziqlpSjpay/Wgp246UsPd4ORabmsPY10OHAGyONMTqTX3cmHk3JcngLkmSdDEJIThaXMO2IyVsP1LKtqMlZBZWA2DQakju7M89I7ozOCaAgV0D6ORlaLIsNehTF/htdoFB1/o94jK4S5IkARn5lbz2v4NsO6J2tQD4e+oZ1DWAaQM7MzgmgKQof4x6rdNlKoqCTts2F3WRwV2SpEtecVUtd/17G9VmK1fGhzIwJoDBMYH0DPFBo+mYV9ySwV2SpEuaxWbnj4t3UFRVy/KZl9En2r+tq+QSMrhLknRJm/vf/WzJKuH16cluE9hBjnOXJOkStmxbNp9sOsofRnTjpv7RbV0dl5LBXZKkS9KOY6U8981eRsYG86eJCW1dHZeTwV2SpEtOfoWJmYtSCPc38tYt/dFp3S8Uut8rkiRJakKt1cbMz1KoqrWy4M6BTY5T78jkAVVJki4ZQgj+/M0+dh4r493bBpAQ3rpnibYl2XKXJKldK6gw8c3O42zOLL7gshZtPsrS7dnMurInE/tEuKB27ZdsuUuS1K6YLDa2HSlhQ0YR6w8Wkp5XWbdsePcgnpwQx8CugS0ud3NmMS+s3M/YhFAeuyrOlVVul2RwlySpTQkhyCioYv3BQtZnFLEls5haqx2DVsOgmAD+NDGBy3sEs+1ICf9ad4gp727iivgQnhgfT1KUc+PSj5ed5MHFO+gS5MXrM/p12LNOW0IRovHMZa1p0KBBYvv27W2ybUmSXMNqs3O0pAa7XaDVKOi1GrQaBZ1GTXOr02pOT2tOpb5VKKk28+uhIjYcLGRDRhF5FSYAeob6MDI2mFFxIQztFoiX4cz2Z43Zyse/HeH9XzIpP2lhYlI4j42LIy7Mt9E6njTbmPrebxwrruGbhy6nR4hPq+6T1qYoSooQYlBz68mWuyRJThFCcKykht055ezOLmNPThmpx8sxWewtKkenUbA60uH6e+oZ0TOYUXHBjIgNIaqTZ5PP9TLo+OOYntw+rCsLN2Sx8NcsftiXx439onhkbCwxwd7n1PlPX+1hf24FC383qMMH9paQLXdJkhpUWFlbF8R35ZSzJ6eMshoLAB46DUlR/vSN9icx0h+jXoPNLrDYBDa7HatdTW1rtQmspx7bBFa7+tjLoOPynsH0ifJHewFdJCXVZt5ff5hPfjuCxSa4eVA0D10ZW/cjsWD9Yf62Kp3/mxDPg1f0dMl+aWvOttxlcJckCVDHfy9PyeHXjCJ2Z5dxolztKtFqFGJDfejXuRN9ozuR3NmfuDBf9O3oxJ+CChP/WneYz7ccA+DWoV3o17kTjy/bxcSkCN6+tT+K4h797DK4S5LkFIvNzvKUHN5ancGJchNdAr1I7tyJ5Gh/kjt3IjHS75y+7/bqeNlJ3lqdwRcpOdjsgoRwX77642VO1d9aWoo5KwtbRQX2ykpslZXYKyqxV1Viq39/apnjXlit6g+HVoui0Zxxj0ZzzjxFoyF41kP4X3vteb1G2ecuSU4SdjvCbFZvtbUIIdCFhLRKS08IgeX4CUz792GvrDq94NS26m+z/qSigEaDxtcXXadOaPz90XbqhNbPD0Xr/MUjTrGbzZgLi1i7KY3vftmLpbCY6UYLI4M1hJcBZcBuOwhBGYJSuwAhwG4HBEIIODVPCBSDAcXogcbDiGI0ojF6oBg81HlGI4qHY56HEcXDgKLXg9WKsFgQVivCUm/aajm9zGJ1LLeg9ffHEBODoVs39BHhDb7uqE6e/H1KX2aO7sEXKdncMqRLg4HdVlWNaf8+TKl7Obk3FdPefViysxvcV4pej8bPD62vLxpfX7S+vugiItD6+qDx8UUxGMBuQ9jsYLMh7I57YQebHWG3gc1+eh27HV1AQIvfs5a6JFruwmzGVlWFNiCgzf+aqV/u45j276c2PR17zUl0IcHoQkLOuGn8/C5qXYUQWHJyOLl7D6bUPVjyC9QFCvXqoZwZhBSlLgCp65wRjc68P2Ne/Q1TFyAQQv1CnDWvLpicmm+3q+vVCzh1z7PbQdgR9ZdZLNjNtY4AblEDuCOQ2y0WsFjO2R8aHx884uIwJsTjERev3sfGovH2PmfdpvapNS+Pk3v3Ytq7D9PevZj27cNWVuZ0Gc1SFDXwdHIE+1NB3zGt6PRYi4uwFRVjLSrCWlyMtbAQe0VFg8Vp/PzQeHqq75VGQUH9UTnjsXJ6nuLoL7ebzQhTLcJkwl6r3rcmxWDA0LVrXbA3xMQ4pmPOCZz22lpq09M5mboX0141mJsPZzo+W6CLjMAzqQ/GPkkY4+PRduqExscXrZ8azDUeHq36WlpKttwdTAcPkj1zJtYTuShGI/rISPUWEYE+KvL048hIdGFh59UKaoywWqnNzKQ2LQ3T/jRMaWmY0tNPf7E0GhSDocEvguLhgS74rKAf6rgPC0cfHoYuPBytz/kd/beWlqof9N17OJm6B9OeVGylpeq2HftJfRHi9L1jWlAv0J61rP5zBPXnnbmsbrrej4SiaOo9dsxDOWueov69PRVgNMrp52k0dS3c+sFI4+WJtlMntXVpMKgtR4MBjcGj3jwPFIPe0QoTmDMPY0o/QPm3K7BXVzveFAV9l84Y4+LxSIjHGB+PR0IC+qgodZ8WFGDat88RQNSAbispUZ+r1eIRF4fPVWPxTErCmJiILjDwrF1y7j6sv7+E1Ya9sgJbeTm2sjLHrd50eTm24hLMmVnYysqwV6n/DDReXmhDgtEFB1MeGkWKZ2cO24wYgoO44rJeDOnfE0NoCNrgYDQG1+RZEUKoLW+TCbvJpP6gmkzYTbUIcy3CYkXR61B0OtDpUPR6FJ1enafXo+jUZYpeD47HtpISzFlZ1B45gjnrCOYjR6g9dIjKtWvBaq3b9qkWvj4qEvORo5gyMup+wLVBQXgmJeF39UQ8+yRhTEpCFxTkktfc3rh1y71q40aOP/IoGk9PAu+6C2thIZYTJ+pudV+8U7Ra9GFharCPikTj7dP4X0wPA4rRiOLhmGfwQFgs1B5IVwN5ejq1Bw8iatVrMSoeHnjEx2Ps1Uu99e6FR1wciocH9upqrAWFWAubvjXU2tJ4e6MLD0cfHo4uPAx9mOM+PFz9EYgIR/HwoDYtjZN7Ujm5Rw3mlqPqgScUBY+ePTD27Ytn32Q8+/bBIzZW/dJJdf+0ag8cUN/TAwcxHUjHciy7LvBqvL1RPD2xFRWpT9Jo8OjZE2NSEsbE3ngmJeERH4/GaLy4dXd0cyhGIxsPFfPPnw6wK7uMLoFePDYulknJURc0UqW9EFYrlpwcNejXC/yW48fRd46ua5V79umDLjy8zf+9X6hL/oBq2fLl5M75Kx7du9P5/ffQR5ybR8J+8iSW3Fwsx0+cEfQtJ05gzc3FXl19Xn8xNX5+ZwRxY69eGLp1u+CAaTeZ1L/W+flYcvOw5udhycvHmpeHJS8Pa14e1qKiM1vHZ9GFhuKZ3Bdjn7549u2LMSkJrY/zXQ2Syl5dTe2hQ5jSD1B74AD2mhqMvXurAb1Xgtq10YitWSX8+du9aDUK43uHMz4xjIRw31YJOtuOlPDPHw+wJauECH8jD4+NZerA6HY10kVqGZcGd0VRrgbmA1rgQyHE389a3gX4BOjkWOdPQohVTZXZWsFd2O0UvjGf4gUL8B4xgqg3Xj/vrou6Mhv7i1lrRtSenoei4BEbhz4qss1aB8JiUf+hOIK9JS8fe1UVHgnxePbtiz48vE3qJak5U1796QAf/ppFdIAnYb5GUo6VIgR0CfRifO8wxieGM7BrwHm1qIUQZJecZOuRErYfKWHbkRIOF1YT7OPBQ1f0YMaQLhj1rut2lNqGy4K7oiha4CAwDsgBtgG3CCH211tnAbBTCPGuoii9gVVCiJimym2N4G6vrSX3mWeoWPU9nW6+mfDnn1P77CSpBYQQLv9x3p1dxhNf7OZQQRW3D+vCMxN74e2ho7Cylp/T8vlpXx4bDxVjttkJ8jZwVa8wxieGcXnP4EYDss0uSMutYNuRErYfKWXbkRIKKtVuQD+jjkExgYyMDWbG4C54GmRQdxeuPKA6BDgkhMh0FLwEuAHYX28dAZxKjOwPnGhZdS+ctbSUnD8+yMmdOwl98gkC77mnw/etSReXEIIPN2Tx5uoMxieGc8+IbvSOvLB832arnbfXZPDOusOE+Hjw6e+HMCoupG55iK8Htwzpwi1DulBVa2XdgQJ+2pfPqtRclm7PxsugZUx8CON7h3NZjyAOF1az/UgJW4+UsPNYGVW16oHESH8jw3sEMSgmkMExAcSF+l4SybGkxjnTcp8KXC2E+IPj8R3AUCHEQ/XWiQB+AgIAb+AqIURKU+W6suVem5VF9v0zseblEfnKy/hdfbVLypUuHdW1Vp76cg/f7cmlX+dOHMyvpMZs4/KeQfxhRHdGx4W0OFim51Xw+NLd7M+tYPKAKP5yfSL+ns79kzRb7WzKLOanfXn8b39+XYsc1IFA8WG+DIoJYHBMIINiApvNySK5D1e23Bv6RJ/9i3AL8LEQ4lVFUYYDixRFSRJCnJFRSFGU+4D7ALp06eLEpptXs307OQ8+BFotXT75GK/+/V1SrnTpyCqq5v5F2zlUUMWfJiZw/6juVJy08vnWY3z8WxZ3f7yNHiHe3DOiO5MHRDXbb22zC95ff5jX/3cQf089798xkAmJLTvWYdBpGB0Xwui4EF68IYldOWVsP1JCz1AfBnYJxN9LdjdKTXOm5T4cmCOEmOB4/AyAEOKleuvsQ23dZzseZwLDhBAFjZXripZ7+cqV5D47G310NJ0XvI+hc+cLKk+69Py8P5/Hlu5Cp1V4+9YBXN4z+IzlZqudVam5fPhrJnuPVxDobeD2YV25Y1hXQnzPPbkls7CKJ77Yzc5jZUxMCmfujUkE+bSvk2Ckjs2VB1R1qAdUxwLHUQ+o3iqE2Fdvne+BpUKIjxVF6QWsBqJEE4VfSHAXQlD83nsUzn8Tr8GDiX7rTbSdOp1XWdLFV1RVy8cbj/BzWj7dgr0deUw60SfaHx+PizO+3mYXzP/5IG+uOUSfKH/evX0A0QFeja4vhGBLVgkfbshidXo+eo2GG/tHcs+I7sSH+2K3Cz7ZdISXf0jHQ6flhRsSmZTcdqOmJPfl6qGQ1wBvoA5z/EgIMU9RlBeA7UKIFY4RMh8APqhdNk8JIX5qqszzDe6FpVUcm/08Xmt+wP+GSYS/+KLLzqqTWld2SQ0fbMhk6bZszDY7Q2ICyS03caykBlD7knuG+NA3uhP9OvvTN7oTCRG+eOhcO9KjrMbMo0t3se5AIdMGRvPijUktGiKYWVjFvzce4YuUbEwWOyNjg7HY7GzOLOGK+BD+PqUvYX4X94Ql6dLhticxrXrkz3T78Qu875tJ58celi2ji8RisZCTk4PpPHKGWGx2Kk1WTpptoICXQYuPh67uRBq7XWC22THb7FisdsxWOzbHx1JRQK9VMGg1GHQaDFoNugs4Acdis1NcZcYmBJ089XhfwD8Fu11QbbZSVWtDCIH/BZbXFoxGI9HR0ejlkOEOw21zy0Q9cB8vlnox/cqpdJGB/aLJycnB19eXmJgYp39Qq2utFFbWYjFZ8FcUunkbCPbxwKBrOjgLIbDY7NSYbZy02NR7sw27cGSr0Wrw9tDh46HF20OHQatxqk6lNWaOl54kPEihS6CXywKx/VQagg72eRRCUFxcTE5ODt26dWvr6kgu1uGCe1JcFPt7DmBDRhE39Itq6+qcN5td8PmWo/h7GZiUHNnW1WmWyWRyKrALIah0BPXqWis6jUKYn5Egb4PTLW5FUTDotBh0Wk4dSRFCUGu1U11rpbrWSpXJSlmNGQC9VoOPh64u4BvO6saxC0FeuYmiqlq8PXR0CfRy6en3HS2on6IoCkFBQRQWFrZ1VaRW0OGCu1ajcHnPYDZkFLbKmYQXQ05pDY8v283WrBK8DVquiA/B19h+/hb/driItekFjmCp3mI9bFSarGg1ChpFQaMBraKgcTwWQlB+0kJBZS0miw29VkOkvycB3gaXJKdSFAWjXotRryXIx6Mu2Fc5gn2lyUqpI9gb6lr2Oox6DSfKTFSbrQT7eBDub+ywwbg1dMTvj+ScDhfcAUbFBvPdnlwyCqqavOp5eyOE4Ksdx5mzYh8CmDm6B+/9cphvdh7njuExbV09QO2TfmLZbvIrTNjrHY75YFIER4qrG3yOoigoqC1kD52W6AAvOnnpWzWI1g/2wY5gb3K07KtMVipMlrpgr1HUbphOXvLAu3Tp6JDBfUSsevr2+oOFHSa4l1abmf1NKqtS8xgcE8BrN/cjOsCTjYeKWLT5KLcP69ouWlGrUnPJLTex8HeDGBMfSrVZbRkXZmfSM8QHmxDYhVAvLCMEdrtwzAMfDy1+Rn2bvA5FUfDUa/F0BHsfHx8KS8qoMdvw9tA1ORrmrrvu4rrrrmPq1KkXscanzZ49m08//ZTS0lKqqqqaf4IkOaFD5v2M6uRJjxBv1mcUtXVVnPLLwUImvLGe/+3P56mr41ly33A6B3qhKAp3DOvKwfwqtmaVNF9QKxNC8MGGTLqHeHNFfChajYKfUU+Evyd6rQYvDx2+Rj3+ngYCHQdHQ/2MRPh7EtXJE39PQ7v4gTrF06AjyMej3WdCvP7669m6dWtbV0NyMx2y5Q4wKi6Ez7ccw2Sxtdsv70mzjb9/n8Ynm44SG+rDR3cNJinK/4x1rk+OZO53+1m0+ShDu7ftFWE2Z5aw93gFf7upT5N5VP66ch/7TzR8mbbz1TvSj79cn9jo8qeffpquXbvyxz/+EYA5c+agKArr16+ntLQUi8XC3LlzueGGG5rdlhCCWbNmsWbNGrp160b94cApKSk8/vjjVFVVERwczMcff0xERARjxoxh6NChrF27lrKyMhYuXMjIkSPZt28fd999N2azGbvdzpdffklsbCyfffYZb775JmazmaFDh/Kvf/0LbSNX+Ro2bJjT+2nlypXMnTsXs9lMUFAQixcvJiwsjKqqKmbNmsX27dtRFIW//OUvTJkyhR9++IFnn30Wm81GcHAwq1evdnpbUsfWIVvuAKNiQ6i12tl+pLStq9Kg1JxyrntrA59sOsrvL+/GylkjzgnsAJ4GLdMGdeaHvXkUVLTudSebs/DXTIK8DUwe0P5GIc2YMYOlS5fWPV62bBl33303X3/9NTt27GDt2rU88cQTOHPextdff82BAwdITU3lgw8+4LfffgPUsfyzZs1i+fLlpKSk8Pvf/57Zs2fXPc9qtbJ161beeOMN/vrXvwLw3nvv8cgjj7Br1y62b99OdHQ0aWlpLF26lI0bN7Jr1y60Wi2LFy92yX4YMWIEmzdvZufOncyYMYNXXnkFgBdffBF/f39SU1PZs2cPV155JYWFhdx77718+eWX7N69my+++MIldZA6hg7bch/aPRC9VmFDRiEjYoObf8JFYrXZeXfdYeavziDYx4PP7hnabP1uG9qFhb9msWRbNg+Pjb1INT3T4cIqfk4r4JGxsc3+E2qqhd1a+vfvT0FBASdOnKCwsJCAgAAiIiJ47LHHWL9+PRqNhuPHj5Ofn094MxckWb9+PbfccgtarZbIyEiuvPJKAA4cOMDevXsZN24cADabjYh6V/CaPHkyAAMHDuTIkSMADB8+nHnz5pGTk8PkyZOJjY1l9erVpKSkMHjwYABOnjxJaGioS/ZDTk4O06dPJzc3F7PZXDc+/eeff2bJkiV16wUEBLBy5UpGjRpVt06g45qt0qWhwwZ3L4OOQV0DWZ9RxDNtXRmHo8XVPLZ0FzuOlXFd3wjm3pjk1AiN7iE+jIwN5vMtx/jjmB4XdAbm+Vr4axYGnYY7hne96Nt21tSpU1m+fDl5eXnMmDGDxYsXU1hYSEpKCnq9npiYGKfPoG3o2IAQgsTERDZt2tTgczw81ARgWq0Wq+OCzLfeeitDhw7lu+++Y8KECXz44YcIIfjd737HSy+91GA5F2LWrFk8/vjjTJo0iXXr1jFnzpy6up/9mjrqUGHJNTpstwzAyLhg0nIrKKhsm+6Mwspa1h0o4J21h3hw8Q4mzt9ARkEV82f0461b+rdo6N0dw7qSV2Hi57RGE2m2muKqWr5MyWHKgCiC23EGwxkzZrBkyRKWL1/O1KlTKS8vJzQ0FL1ez9q1azl69KhT5YwaNYolS5Zgs9nIzc1l7dq1AMTHx1NYWFgX3C0WC/v27WuqKDIzM+nevTsPP/wwkyZNYs+ePYwdO5bly5dTUKC+lyUlJU7XrTnl5eVERandZp988knd/PHjx/P222/XPS4tLWX48OH88ssvZGVl1dVDunR06OA+yjEk8tdWHjWjXpuyhh/25vLqTwe4+99bGTLvZwbP+5m7/r2Nf/x4gN05ZYzrHcYPj47ihn5RLW4xXZkQSqS/kc82uyYItMRnm49Ra7Vzz4juF33bLZGYmEhlZSVRUVFERERw2223sX37dgYNGsTixYtJSEhwqpybbrqJ2NhY+vTpwwMPPMDo0aMBMBgMLF++nKeffprk5GT69etX1x/fmKVLl5KUlES/fv1IT0/nzjvvpHfv3sydO5fx48fTt29fxo0bR25ubqNlPPXUU0RHR1NTU0N0dHRda7whc+bMYdq0aYwcOZLg4NPdfc899xylpaUkJSWRnJzM2rVrCQkJYcGCBUyePJnk5GSmT5/u1P6R3EOHSxxWn90uGDzvZ0bFhfD69H4uqplq3YECNmQUse9EOftPVFBhUv+GazUKPUK8SYz0JzHSj96RfiRG+Lvk4glvr8ngnz8dZPUTo+kRcmEX9XaWyWJjxMtr6BvdiY/uGtzoemlpafTq1eui1Em6uOR727G4beKw+jQahRGxwWzIKMJuFy67ZuSenDLu+vc2jHoNCeF+XJ8cSWKkP70j/UgI9221oZc3D+7M/NUZfLb56EU7aPnNzuMUVZn5w0iZOEqS3EmHDu4AI2ND+HbXCdLzKi/4YsanLNmWjVGvYcuzVzl9zUtXCPU1cnVSBMtTcvi/CfF4GVr37bHbBR/+mkVipB/D23iMfWtITU3ljjvuOGOeh4cHW7ZsaZP6DB06lNra2jPmLVq0iD59+pyz7rx5884Zujht2rQzhmZKUlPcILir/Y4bMgpdEtxrzFZW7DrBtX0iL2pgP+WOYV1ZufsEK3adYMYQ11xntjG/HCzkUEEVb0zv55ajKvr06cMYT2IjAAAgAElEQVSuXbvauhp1WvKjMnv2bBnIpQvSoQ+oAoT5GYkP82V9hmvSlv53Ty5VtVZmDGmb67EOjgkgIdyXTzcddeqEnAvxwYZMIvyNXNs3ovmVJUnqUDp8cAe19b4tq1S90s8FWrotm+4h3gzqGuCCmrWcoijcPqwr+3Mr2Jld1mrb2XeinN8OF3PXZTEuzW0uSVL74Bbf6pFxIZhtdrZkFV9QORn5laQcLWXG4M5t2k1xY/8ofDx0LNrUesMiF27IwtugbfWuH0mS2oZbBPchMYEYdBo2XOB496XbstFrFSYPiHZRzc6Pj4eOyQOi+G5PLsVVtc0/oYXyyk2s2H2Cmwd3bpPjCpIktT63CO6eBi1DuwWy4QL63WutNr7ckcO43mHt4izN24d1xWyzs2x7jsvL/vi3I9iF4PeXu+/wRx8f588TuOuuu1i+fHkr1qZxNTU1XHvttSQkJJCYmMif/vSnNqmH5H7cIriD2u9+ML+KvPLzS0Xwv/35lNZYmD64fXRTxIX5Mqx7IIu3HMVmd92B1epaK59vOcrEpAg6B3q5rFzp/D355JOkp6ezc+dONm7cyPfff9/WVZLcQIcfCnnKyNgQIJ31GYXcPKjlI12WbssmqpMnI3q2nwyTdwyL4cHPd/DLwQKuTAhzSZnLtmdTYbJe2ElL3/8J8lJdUp864X1g4t8bXeyu+dy9vLy44oorADX9wYABA8jJafzfmsznLjnLbVruCeG+BPt4nFe/e3ZJDRsyirh5UGeXXMzZVcYnhhHi68GnLjqwarMLPtqYxaCuAfTv0jajgc7XpZDPvaysjJUrVzJ27NhG15H53CVnuU3LXVEURsUGs/ZAQYtTEXyxPRtFgWmD2vZA6tn0Wg23DOnCW2syOFpcTdcg7wsq78d9eWSXnGT2Nb0vrGJNtLBbi7vnc7dardxyyy08/PDDdO/eeAI3mc9dcpbbtNxBTQFcWmNhXwsuAWd1HLQcHRdCZCfPVqzd+bllSGc0isLnW45dcFkfbMika5AX43q7povnYjuVz33p0qXn5HPftWsXYWFhLsnnvmvXLnbt2kVqaio//fRT3fLG8rmvWLECT09PJkyYwJo1a+ryuZ8q58CBA01megS47777iI2N5dFHH21yvVmzZvHQQw+RmprK+++/X/d6ZT536WxuFdxH9FRTALfkbNX1GYXkVZiYMbhtzkhtToS/J+N6hbF0ezYmy/mfpJVytISdx8q4Z0S3dtX11BLums/9ueeeo7y8nDfeeKPZust87pKz3Cq4h/h60DvCr0VDIpdszSbYx8DYXu23NXvn8K6U1Vj4bk/jOcGb88H6LPw99Uwd2L66nlrCHfO55+TkMG/ePPbv38+AAQPo168fH374YaPbk/ncJWd16HzuDXnp+zQ++jWLXX8ej7dH04cUCipNDH9pDX8Y2Y1nJrbffNZCCMa+9gu+Rj3fPnh5i59/tLiaMf9cxx/H9OD/JjgXAM8mc367L/nedizO5nN3q5Y7qFdnstgEmzObT0WwPCUHm10w/TyGTl5MiqJwx7Cu7M4uY09Oy/PNfPRrFnqNht8Nj3F95SRJapfcZrTMKQO7BmDUq6kImupqEUKwdFs2Q7oF0v0iXfXoQkweEM0rPxzgs81HeWVqp2bXrzFbKaio5UT5SZZtz2FSv0hC/YwXoabth8znLl3K3C64G/VahnYLavag6ubMEo4W1/DoVbEXqWYXxt9Tz439I/lqx3FuH9aVSpOVgkoTBRW1FFQ6bhUmCh3TVbXWuucadBruHdm+r4/aGmQ+d+lS5nbBHdRUBHO/SyOntIbogIZPsV+67Ri+Rh0TkzpOLvPbh3XlP1uzmfT2xjPme+q1hPp5EOrrQa8IP0bFeTgeGwn19aBHqA9R7XCYpyRJrcctg/vouBDmfpfGrxlFDaa0La+xsGpvHjMGd26166G2hsRIfz64cxA1ZqsauB0B3cdDJ8czS5J0BrcM7j1DfQj3M7I+o7DB4P71zhzMVjvT2+nY9qZ01BOQJEm6uNxutAyoo0tGxgbza0bRORkVhRAs2ZZN32h/EiP926iGkiRJrcstgzuoV2eqMFnPGTq4J6ec9LzKDtlql5zXUfK5A1x99dUkJyeTmJjIzJkzsdku/HKRkuRUcFcU5WpFUQ4oinJIUZQGryagKMrNiqLsVxRln6Ion7u2mi03omcwisI5WSKXbMvGU69lUnJkG9VMks60bNkydu/ezd69eyksLJTZGyWXaLbPXVEULfAOMA7IAbYpirJCCLG/3jqxwDPA5UKIUkVRmk6BdxEEehtIivRnQ0YhD49VhztW11pZses41/aNwNcoLy93vl7e+jLpJekuLTMhMIGnhzzd6HJ3zecO4OfnB6iZIc1mc5MHxz/44AMWLFiA2WymZ8+eLFq0CC8vL/Lz85k5cyaZmZkAvPvuu1x22WV8+umn/POf/0RRFPr27cuiRYua3T+Se3Cm5T4EOCSEyBRCmIElwNnfoHuBd4QQpQBCiALXVvP8jIwNZsexMipNFgC+25NLtdnGLUNkl0xH4+753CdMmEBoaCi+vr5MnTq10fUmT57Mtm3b2L17N7169WLhwoUAPPzww4wePZrdu3ezY8cOEhMT2bdvH/PmzWPNmjXs3r2b+fPnN7tvJPfhzGiZKCC73uMcYOhZ68QBKIqyEdACc4QQP5xdkKIo9wH3AXTp0vqXsxsVF8K/1h3mt8PFTEgMZ8m2Y/QM9WFAB7tQRXvTVAu7tbh7Pvcff/wRk8nEbbfdxpo1a+rqcLa9e/fy3HPPUVZWRlVVFRMmTABgzZo1fPrpp4Caktjf359PP/2UqVOn1iUYk/ncLy3OBPeG/iOe3TzSAbHAGCAa2KAoSpIQ4oyjmUKIBcACUBOHtbi2LTSgSwBeBi0bMgrpFuzNjmNlPHdtLzkmvIM6lc89Ly/vnHzuer2emJgYl+RzP5Xy92yN5XMfOnQo3333HRMmTODDDz+sy+f+0ksvtej1GY1GJk2axLfffttocL/rrrv45ptvSE5O5uOPP2bdunWNlifzuV/anOmWyQHq92NEAycaWOdbIYRFCJEFHEAN9m3KoNMwvHsQGzKKWLI1G71W4ab+UW1dLek8uWM+96qqqrp0wFarlVWrVjWZuriyspKIiAgsFssZXT1jx47l3XffBdR/HBUVFYwdO5Zly5ZRXFxcVw/p0uFMcN8GxCqK0k1RFAMwA1hx1jrfAFcAKIoSjNpNk+nKip6vkbHBHC2uYcm2Y4xPDCfIx6OtqySdJ3fM515dXc2kSZPo27cvycnJhIaGMnPmzEa39+KLLzJ06FDGjRt3xuudP38+a9eupU+fPgwcOJB9+/aRmJjI7NmzGT16NMnJyTz++ONO7R/JPTiVz11RlGuAN1D70z8SQsxTFOUFYLsQYoWi/vd7FbgasAHzhBBLGi+x9fK5n+1wYRVjX/0FgEX3DGFkbEirb9MdyZzf7ku+tx2Ls/ncnUo/IIRYBaw6a96f600L4HHHrV3pHuxdlzTr8h7BzawtSZLkHtwyt0x9iqLw+vR+6LQKmg567VDp/HTkfO4PPvggGzeemf3zkUce4e67727VOkruw+2DO8CQbnII2KWoI+dzf+edd1qxJtKlwG1zy0iSJF3KZHCXJElyQzK4S5IkuSEZ3CVJktyQDO6SW+pI+dxPmTRpEklJSW1dDclNyOAuSe3AV1991aIfJElqziUxFFJyrby//Y3aNNfmc/folUD4s882utyd87lXVVXx2muvsWDBAm6++eYm6y7zuUvOki13qUNw53zuzz//PE888QReXl7N1l3mc5ecJVvuUos11cJuLe6az33Xrl0cOnSI119/va7Mpsh87m6gJAsCu7X6ZmRwlzoMd8znvmnTJlJSUoiJicFqtVJQUMCYMWMazdMu87l3cPn74IMr4ao5MOyBVt2U7JaROgx3zOf+wAMPcOLECY4cOcKvv/5KXFxckwFb5nPvwMzV8MVdYPSHpCmtvjkZ3KUOwx3zubeUzOfega36PyjKgMkfgE/Tl110BafyubeGi5XPXXINmfPbfcn39iLY9R/4ZiaMfhquuLBjVs7mc5ctd0mSpNZUeBC+exxiRqrB/SKRB1QltyXzuUttznJS7WfXe6rdMRotn6d9zjXdrqGTsVOrbloGd8ltyXzuUpv74U9QsA9u+xL8Ilh9dDUvbX0Jq93KnYl3tuqmZbeMJElSa0hdDikfw4jHIPYqKs2VzNsyj4TABG7tdWurb1623CVJklyt+DCsfBQ6D4Ur1LOc5++YT7GpmLeufAudpvVDr2y5S5IkuZK1Vu1n1+pg6keg1bOzYCdLDyzltl63kRiceFGqIVvukiRJrvTTc5C3B25ZAv7RmG1m5vw2h0jvSB7q99BFq4ZsuUtuqSPlcx8zZgzx8fH069ePfv361Z3ZKnVA+1fA1gUw7EGInwjAR3s/IrM8k9nDZuOlbz45nKvIlrsktQOLFy9m0KBmz0uR2rPSI/DtQxA5QM0dA2SWZ7JgzwImxkxkVPSoi1odGdylFtuw7CBF2VUuLTO4sw8jb45rdLk753NviZUrVzJ37lzMZjNBQUEsXryYsLAwqqqqmDVrFtu3b0dRFP7yl78wZcoUfvjhB5599llsNhvBwcGsXr36gusgNcBqhuW/V6en/Rt0BuzCzl9/+ytGnZGnhjx10asku2WkDsGd87kD3H333fTr148XX3yxydcwYsQINm/ezM6dO5kxYwavvPIKoOac8ff3JzU1lT179nDllVdSWFjIvffey5dffsnu3bv54osvmt030nla/Vc4ngI3vAUBMQB8nfE1Owp28OSgJwn2DL7oVZItd6nFmmphtxZ3zecOapdMVFQUlZWVTJkyhUWLFnHnnQ2f4JKTk8P06dPJzc3FbDbTrZuaF/znn39myZIldesFBASwcuVKRo0aVbeOzOfeSg78AJvehsH3Qm/1n2PRySJeTXmVweGDuannTW1SLRncpQ7DHfO5A0RFRQHg6+vLrbfeytatWxsN7rNmzeLxxx9n0qRJrFu3jjlz5tTV/ezXJPO5XwTlOWpCsPA+MH5u3ey/b/07tdZa/jzsz232HshuGanDcMd87larlaKiorrt/fe//yUpKanR7ZWXl9f9GHzyySd188ePH8/bb79d97i0tJThw4fzyy+/kJWVVVcPyYWEgK9ngs0C0z4BvRGAX7J/4ccjP3Jf3/uI8Y9ps+rJ4C51GO6Yz722tpYJEybQt29f+vXrR1RUFPfee2+j25szZw7Tpk1j5MiRdZfPA3juuecoLS0lKSmJ5ORk1q5dS0hICAsWLGDy5MkkJyczffp0p/aP5KTd/4EjG2DCPAjqAUC1pZq5W+bSs1NPfp/0+zatnsznLjlF5vx2X/K9PQ81JfD2IAjsAb//ETRqO/nlrS+zOG0xn078lH6h/Vpl0zKfuyRJUmv5eQ6cLIPrXq8L7KmFqSxOW8zN8Te3WmBvCXlAVXJbHTmf+7x5884Zujht2rQzhmZKbeTYFtjxCQx/CMLV4yMWu4U5m+YQ4hXCowMebeMKqmRwl9xWR87nPnv2bBnI2yObVb2qkl8UjHmmbvYn+z7hYOlB3rjiDXwMzqe+aE0yuEuSJDlry3uQvxduXgQeahA/VnGM93a/x9guYxnbZWwbV/A02ecuSZLkjPIcWPs3iB0Pva4H1HMJXtj8AnqNnmeGPNNMAReXU8FdUZSrFUU5oCjKIUVR/tTEelMVRRGKosgMSJIkuZcf/gTCBtf8AxwnJn196Gu25G7h0QGPEuYd1sYVPFOz3TKKomiBd4BxQA6wTVGUFUKI/Wet5ws8DLTN0SpJkqTWcvAnSFsJVz4PATFUmit5I+UNlh1cxoDQAUyLn9bWNTyHMy33IcAhIUSmEMIMLAEaSr33IvAK4Nz535LUijpSPnez2cx9991HXFwcCQkJfPnll21WF6kB5hpY9SQEx8FlD7P66Gpu/OZGlmcs547ed/DuVe+iUdpfD7czB1SjgOx6j3OAofVXUBSlP9BZCPFfRVGedGH9JMntzZs3j9DQUA4ePIjdbpdpAtqbDa9C2VEKZizibxueYvWx1cQHxDP/yvkkBTeeKqKtORPcG8p6U3daq6IoGuB14K5mC1KU+4D7ALp06eJcDaV2Z+3HCyg4munSMkO7dueKu+5rdLk753P/6KOPSE9PB0Cj0ZyRVuBsMp/7RVZ4APvG+SxPGMPrO17CYrfw6IBHuTPxTvQafVvXrknO/JfIATrXexwNnKj32BdIAtYpinIEGAasaOigqhBigRBikBBiUEhIyPnXWrrkuGs+97KyMgCef/55BgwYwLRp08jPz2+07jKf+0UkBJnfPczdESG8WJtJYlAiX036inv63NPuAzs413LfBsQqitINOA7MAG49tVAIUQ7UNTUURVkHPCmEkIlj3FRTLezW4q753K1WKzk5OVx++eW89tprvPbaazz55JMsWrSowfVlPveLw2Kz8OHPj/IBx/E0evPi8Oe4occNHSqFcrPBXQhhVRTlIeBHQAt8JITYpyjKC8B2IcSK1q6kJIF75nMPCgrCy8uLm25SL+gwbdo0Fi5c2Oj6Mp9769tVsIs5G5/ncMURJtqNPDXte4K9O15Pg1OHeIUQq4QQcUKIHkKIeY55f24osAshxshWu9Qa3DGfu6IoXH/99axbtw6A1atX07t370a3J/O5t54qcxVzN8/lzu/vpKYqj3fyi3hl4sIOGdhBnqEqdSDumM8d4OWXX2bOnDn07duXRYsW8eqrrza6rszn7npCCP539H/c8M0NLDuwjNs6X8U3mYcY1ed3EJHc1tU7bzKfu+QUmfPbfV3K7+2JqhP8bcvf+CXnFxICE/jzkNn0+XoWVBfCg1vB6NfWVTyHs/ncZeIwSZIuOVa7lcVpi3ln1zsAPDnoSW7rdRu6rR9A3h6Y+u92GdhbQgZ3yW3JfO5SQ1ILU3lh8wukl6QzJnoMzwx9hkifSKjIhTXzoMdYSLyprat5wWS3jOSUtLQ0EhIS5OgLNyOEID09/ZLolqk0V/LWzrdYkr6EEK8Qnh3yLFd2uVL9TJtr4NNJkJcKD/xWd03U9kh2y0guZTQaKS4uJigoSAZ4NyGEoLi4GKPR2NZVaVWnDpi+vPVlCk8WcmuvW3mo30OnL6pht8FX90LOdrj503Yd2FtCBnfJKdHR0eTk5FBYWNjWVZFcyGg0Eh0d3dbVaDUnqk4wb8s81uesp1dgr3PzwQgB3z8N6f+Fq1+G3pParrIuJoO75BS9Xl93pqMktWdCCI5WHOXnYz+zYM8CAP5v0P9xa69b0WnOCnm/vQnbPlCvhzpsZhvUtvXI4C5JUodmsVtIL05nR8EOdhbsZGfBTkpM6glbY6LH8OzQZ4nwiTj3ianL4X9/Vg+ejnvxIte69cngLklSh1JlrmJ34W52FOxgV8Eu9hTuwWRT00509u3MiKgRDAgdQP+w/nT3795wIVkb4JsHoOvlcON7oHG/8zllcJckqd3bkruF1cdWs7NgJwdLD2IXdjSKhoTABKbGTaV/aH/6h/YnxMuJVAEFabDkNgjoBjMWg949DyjL4C5JUrtVUFPAy1tf5qejP+Gp86RvSF/u73s//UP70zekL95675YVWHECPpuqBvTbl4NnQOtUvB2QwV2SpHbHZrfxxcEvmL9jPmabmYf6PcTdSXdj0BrOv1BTBSyeBqYyuHsVdHLvCwbJ4C5JUrtyoOQAL2x6gT1FexgaMZTnhz1PV7+uF1ao1QzL7oDCdLh1WYdOCOYsGdwlSWoXaiw1vLv7XRbtX4S/hz8vjXyJa7tde+EnzQkBKx+GzHVww7+g51iX1Le9k8FdktzV5nfhwCpInKwO9/Ps1NY1atT6nPXM2zyPE9UnmBI7hccGPoa/h79rCl87D3b/B8Y8C/1vc02ZHYAM7pLkboSA1S/Ar6+BVzBkrVfPwky4FvrdCt2vAK0OIQQFNQWklaSpt+I0DpQcQCAI8wojzDuMMK8wQr1CCfMOI9wrnDCvMIK9gl12DdH6B0y7+3fn46s/ZmDYQJeUDcD2f8P6f0D/O2D0U64rtwOQwV2S3IndDquehO0LYeBdcO1rkLsbsetzctK+JC3re9J8A0gLiCRN1FJiLgdAQaGrX1eSQ5LRaXTk1+STXpLOL9m/1I0hP0VBIcgzSP0BcAT/IM8gAo2BBBoDCTAG1E37GnzRKOeOIbfZbSw7uIw3d7yJ2WZmVv9Z3J14N3ptvR8Nux1ytkHaCvV2shy8AsErqN7t7Mf1bjnb4LsnoOc4uO51uMRyIsngLknuwmaBr2fC3uUUDruflPgrSU15jbSSNNLL06kM8QF80AE9Ko8zqraWBGMIvbtPJH7gH/Dy73xOkUIIKswV5Nfkk1+dr97X5FNQU0B+dT7HKo+xLW8blZbKBqukU3R0MnY6J+jvKdxDalEqwyKG8fyw5+ni5xi5YrfBsc2w/1s1oFfmgtYAPa6ETl3hZAnUFENVHhTsV6ctNY3vk4hkmPYxaF3zT6MjkSl/pQ6jau8X5OXuJHLUs3h5+LR1ddoNIQQ5pYfY/t2D7CjPYEdgJMcsFQB4aD2ID4inV1AvEgIT6BXUi9hOsRhOlsPe5bDrc/XiFBodxE6AfrdAl+Hq+G+N1uk6mG1mSk2llNaWUnKyhJLaEkpOllBaW0qpqZRiUzGlplJKTCWUmkrx1Hny+KDH1QOmdhsc/dUR0P8L1QWgM0LPq6D3DRA3AYxN9L+ba04H/ZpiqHFMW05Cv9vAp2NeA7Uxzqb8lcFdap7Nql52rCr/9H1VgXqrdtybymDEY5A0xeWbtws73+z4F2/sfpdSrfoXP9QzhM5+Xeji24Uufl3o7NuZzr6d6eLb5XQqVzdlF3YySjPYUbCDlPwUduRtp9BUDIC/1siAyOEMDBvIgNABJAQlNN8/nr9PDfJ7lqnv5ylGf/AMVAO9V6A67eV4XDfdSX2s91Jb2Doj6DzUm9ZDbTE31B1is0DWL6cD+skStYzY8WpAjx0P8ge8QTK4dwS/vgH7vlK/FBq9+kXQGhz3+obn671g0N3g34ppWvd+CSmfnA7kNSVAA58Tg6/aKvIJg+oiKDsG966G8HOvLHS+9hXv42+b5rKneC/9zTamhAwiP2stx0JjyQ6MJrsyh8KTZ6YhDjQG1gX6zn6dSQpKYkDYgJafzdgUIbAV7Gdf2pf8lrOebMWGNnoIWoMXWkWLXqNHq2jRarRoFS06ja7u/tT0qSF+Z38HRb19XX+ZyWZiT+EedhTsoNKsdoOEeYYwoKqMQWWFDBj5LN0H3d9gH7dTbFbIWgfFh9X3/GSJ4770zOnaipaVqzOqgb4u6BugpghM5epnKP5qNaD3GAsGr/Or+yVEBvf2rrYK/hkHfhHqmXI2i+NmVu/tp6atjnsz2K1groKwRLjn59bJiXE8BRaOV/s3Q3uBT6gavL0dQdwnVL15h575RawqhPdGgMEb7lt3wdefLK8t580db/LFwS8IVPQ8XpDL9ZM+QYkbBxteVUeDJE2BmxZQYzeTXZlNdmU2xyqPcaziWN3jvOo8BAKdoiMpOInB4YMZGjGU5JBkjLoW7r+SLI4fWMmmI//jt4rDbNFDhVaLIiDMZkNotNg8O2FDwWq3YhVWbHYbNqHeXCHGL0ZtlYcNYKBnFJFf3Y9SlgPTF0HsOJdso1k2C5wsOzPgW03qZ9RqUk8YstWeNe24nVrH4A3x16gjd9w0t0trkcG9vdv1H/hmJtz9PXS9zPnnHfwRPr8ZhtwP17zi2jrVVsJ7I/lRa6Fw5CNMS/odHloP559/ZCN8ch30vhGmfnReoxPsws5XGV8xf8d8Ks2V3BI2nD9u/g++Qx+Aq186veLG+Wq61t43wJSFjR4wO2k9ye7C3WzN3cqWvC3sK9qHTdgwaAz0C+3HkPAhDI0YSmJw4rndFxW5VB3+mW2Hv+O34n1s0lo4qlfXCVMMXNYpnuHdr2Zoz+sILDwMS25V+3mnLlT7iesRQpwR7K12a5P7of6JOwrqtFbR4qV3/KAWHYJFN6qt31uXtuwzJHVoMri3d59cr3ZjPLyr5UHwh2dg879gxufq2GUXqfnyD/wtdw3f+qrdFxHeETwy4BEmdpvo/F/9Da/B6r/CNf+EIfe2aPt7i/Yyb/M89hbvZUDoAJ7tcz/xn98OvpFqd4/urB+aTe/Aj89CwnXq1ep1zecdqTJXsaNgB1tzt7I1byvpJekIBJ46TwaGDWSofyxxx1PZU7CLTaKKPR4eWBUFTxQG+3Tlsi5XMLznDXTr1P3cMyfLc+A/t6jX4Rz3Alw2q3WG3+Xugc8mq+PZ7/jqkjiVXjpNBvf2rOwYvNFHPWNuzNMtf761FhaOg9Kj8MBGl/S/Z2x+kydT/0WWQc99fe9nYNhAXk95nbSSNBKDEnli0BMMDh/cfEF2O/xnunqq9z0/QWT/Zp9SZipj/s75fHnwS4I8g3h84ONcF3MNyuIp6rC4+3+BkPiGn7zlffj+KYibCDd/cu4PgBPb3p6/nS3Hf2Vr1k9kWtW+bAXo7RHMZZGXMbzn9SSHDXAuaZW5Ws0Tvv9bSL4Vrn+jxXVq0rHNsPhm8PCFO7+B4FjXlS11CM4Gd4QQbXIbOHCguGSte0WIv/gJUXLk/MsoOiTEvEghFl4thNVy3sXY7XaxfMe7YuC/E8Xof/cVv+X8WrfMZreJFYdWiLHLxoqkj5PEQ6sfEpllmc0XWlUkxKu9hHijrxA1pY2uZrVZxdL0peLy/1wukj9JFi9vfVlU1laqC3+dr+6jbR81v72tH6jrLpoihPlk8+vXZ7MJsXOxEP+IFeIvfqLgi7vExowVouRkScvKObvMtS+pdfrgKiEq88+/rPoO/k+IF8OEeHOAEKXHXFOm1OEA24UTMfbSaAsb8IAAABL1SURBVLmnr4LDa04frDx1gLLBacc9Qj2rLWaEa+siBLw1APyi4K7/XlhZu5fC1/fB6Kfhimdb/PRqSzUv/DaHVUd+YGitlb/fsJTgsL7nrGeymvgs7TM+TP0Qk9XE1LipPJD8AEGeQY0XfmwLfHwNxF0N0z+r656w2W1klGWQkp/CisMr2F+8n4FhA5k9dDaxAY5W6Imd8OE4td+63nOblPIxrHwUelyhdlfpPZt/TvZW9bT8EzsgahBMfBmim28QOW3f1/D1A+rZkrf8ByLO3bfNslkg439qbpQDqyC0N9z+lduN3ZacJ7tlTjlZBq8ngbCBh1+9oYb1788afqjVqwcHQ+LVvM+udGwzfDQBbnxXzfNxob5+QP3i/24ldBvp9NPSitN48pcnyak8xoMlZdwz4W20vW9o8jnFJ4t5d/e7LD+4HKPOyB/6/IHbe93e+KiT397C8tNz7B/9KCkhXUnJT2Fn/s66sxk7+3bmwX4Pck23a073X9dWwfuj1AOTD2xUx1I7a+dn8O1D6n64ZYk6IqMh5cfh5zmQugx8I+Cqv0Kfaa1zqbUTu9R+eFMZ3PQ+9J7U/HOEgNxdsHsJpH6hnpDjFQx9p6vdeE2d0CO5PRncT1n/D1gzF+5f37IDT5v+BT8+A39YA9EuTGS0Yhakfkn2zLUsP7KKKJ8obo6/+fzLq62CBaPVvt6ZG8G7idY0ajfckgNL+Me2fxCg8+LlowcZlHQ7XPea05vMLM/k9ZTXWZe9jnDv8P9v787ja7rWBo7/VkJU6UViaCml5qJEiBKUt1VVRcul3Hqr19iW4qqWvLdX6YDqRXu97e0raK/3VhCpqbSqVKmpMURiKqqGmGciQobn/rFOCBI5J9NxTp7v55NPkr332XutbJ6zztprrYfBgYNp/3B7fIwPicmJxJ6OZdOJTWw+vpmY479wxTFuu0qJKgSVC7JfZYMyTlq8cJAN0i6+WV23bbbt867UzI4iST8RJukKrJsCP0+209xDBkPI0LyfLHPpuE3rdmQTtH4bWg7P+NPIxaN2ItG22XBql21o1HwG6vewy9QWwCn06nYa3MFOS/64rn2o1zPStddevQST6kC1/7JrU+SC1KvxbPi0HuHlKvFT8jkEwdf4EtEh4kaXRHYci4FpT9gxw3+ak2k3xsVrF3ln7Tv8cOgHmpcLZuz21ZQqGgD9f3SuG+MWUcej+CjqI3ad3UUt/1oULVSU2NOxJKcmYzDU9K9JUEAdgrZG0jBJCOi/+s4t8R3zIeJlaPEGPDHK5fJcFzsPvu4HFZvAixHgV9yee/kouHDYDtVs8y6UymECCFckJTre2Ofa8fmdPrV/82uXYfcSO0N0/ypAbLnrd3cs0+u9aeBU9mhwhxsjKV5eCpVDXH/98lG2pTd4K5SqnO1ixF+LZ+FvC5m9LYwDV8/gX7g4XWr1oG3ltvRe1pva/rUJeyosZ0kJ0uradhw0fe223bGnYnlz9ZucuHyCIYGDeSl6MT4H10K/H6HcI9m+bKqksmT/EqbHTqeYXzGCygXRqFwjGpRtwB/8HBOZjmyG6W3tWiE9wjN+8zl/GD4PgYBq0HtZzlupO+bDvD72jd3XDw6tg3L1oN343H+O4iwR+6lhxbtQvoHtP9+50E5MK1HJBvT63SGgqnvKpzyCBveUJPhHIPyhvA0W2QmcF4/aIYuN+9mg4KL9F/YTviucRb8tIiE5gXoUoUf8Vdr2j8LPMSsvfHc4YzeOZVKrSbR5KAczDEXsJJq9y6Hv8utDEEWEmTtn8vHmjyl7b1kmPD6B+nvXwPd/tcvBNu6T/Wu6Iu3Np817tjskvdQU+LK9HR/+yhrwfzh3rrlzEcz7M9xTEp74m13T24XFsPLM7iUQ2Q+MD9TpZLtdKjXLmz5/5XW8dijkuSvnZOn+pVkfuHWWHYq2+9tsXee6rweIvP+ASIJzQ+OSU5Jl5cGV0m9ZP6n7ZV0JnBkooatDJeb3FSLvlBBZ+cFNxyelJEnnhZ2lTUQbSUhKyFlZL5+xQxA/aSCSeFFERKZumyp1v6wrg1cMlvOJ50WObBEZEyAS/ieR1NScXc8Vqakic/5bZHQpkYPrb9636kN7r6LDc/+6p/eJXLmQ++fNqauX7ZdSLsLJoZAet577zJ0zCYsNY8OxDYQGh2Y8UiM11X78LVvntmngzhARtp3axrnEc1D1MWTvAvhpNJJ+pIOkfbvxyefgxYNE7IngSPwRyt5bltcDX6dL9S52yOCaifZF9bvfdK1CPoUYGTyS3st688X2L3itwe1dKk671x86h9klAL4Zxo6WQ/gs+jOervw0E1pOwFyLh3m97dowHafkb/ICY+w1j8VAxJ/hlZ/tw99DG2HVeDta5dEXcv+6d2sXhy6QpfKYxwX3tOAXFhvG9tPbmdRq0u2Z0X9dCqd/hc7TXA5gZ66c4b0N77Hi0IobG8uVgRMr7VcWgsoFMSxoGK0rtb6xVomIXUumUrMMuxwa39+YdpXbMWP7DDpV60SF4hVcKvNNKofA4yNJ+GkcI5P2EFA0gLcfe9v25y99C84dgF7fuDbEMLfcU8LOIp3Wxo7P7zINvu5rZ9i2n1jgMuUolZc8ts99TdwaQn8OJTk1mTHNxtC2sqOFLmJHjiScgUGbwdf5968fDv7Au+vfJT4pnoENBtK0fFO74/AvmKXDoVUoJt1aLmkLOqUp7lc848B8OAqmP2lbrg1fyvDaxy8fp+OCjoSUD2Fy68lOlzlDqSm8P7M5c0w805qMoUmtznaI3df94PGR0Do0Z+fPqajpsGSYfYh48Qj0/g4qBru3TEp5CGf73J2KfMaYp4FPAF9gmoiMv2X/MKAvkAycAnqLyEGXS+2CFg+2IOLZCIavHs7wn4az5cQWhjcaTuGD6+zojPaTnA7sF65eYNwv41iyfwm1/WsztvlYqpWqduMA/9qwbipsjYCmw1xvYW6bBYWK2iF4mbi/2P30rdeXKVunsP7o+htvLNmw+uha5ph4el2+RpNVk8D/EfhmmM2w0/LNbJ831zTqDQfX2UxArd/WwK5UHsiy5W6M8QX2AG2AOCAK6CEiO9Md0xrYKCIJxphXgVYicscO1NwaLZOUksSkzZP4965/U690Pf5++jzlT+2DITFOrRO9Om41o9eN5lziOfrX70/fen0zzlyTtkTvi5FQ/UkXCpgIE2vYFGZdwu546NWUqzy34Dn8fP2Y13FetjLMn7lyhs6LOhNQNIDZ1XvjN+dFR5acwnaSU8nb82S6xbUEO667Rtu7YwSLUh7C2Za7M2OvgoF9IrJfRK4Bs4Gb5qmLyI8ikpaldgOQh2mCblbYtzAjgkcwudVkfj+3j64pB/ip3rNZBvb4a/GMXjeagSsGUqJICb5q/xWv1n8184Bat4udqr5+imsF/HWpXXPbiaUGivgW4a3Gb7H/wn5m757t2nWwD4JHrx9N/LV4xrcYj1/tZ+Gx12wC4Y5T7p7ADvaBYq1nNLArlUecCe4VgMPpfo9zbMtMH+DbnBQqO5586Enm+j5EhVQYdHw5kzdPzjQhwsZjG+m8qDPz982nT90+zHl2Do8EZDGRp5AfNHnFtjaPxThfsOhZdpGwKi2dOrxVxVaEVAjhs+jPOHPljPPXASL3RrLq8CqGNBxCjVI17ManPrBrxmexboxSyrs4E9wz6mDOsC/HGNMTaAR8lMn+/saYTcaYTadOncrokOw7tYeKvy7n/yt3o2uNrszYPoM+y/pwMuFGwt+EpATGbRxH3+/7UsS3CDPbzWRo0FDn1ukGCHrZTmVf/7/OHX/pOPy2wg5/dLKFaoxhROMRJCYn8smWT5y7DnYY5oSoCTR5oAk9H+l5Y4ePD/hXcfo8Sinv4ExwjwPSf55/EDh660HGmCeBvwIdReRqRicSkaki0khEGpUpk8tLlq79GArdQ5GmgxjVdBRjm49l19lddF3clfVH1xN9Mpqui7sya/csetbuydwOc6lfxsUMNkVL2tEu2yNt1p2sxMwBSbUzEF1QpUQVej7Sk/n75hN7KjbL45NSkwhdE0phn8K8H/J+9hMkK6W8hjNRIAqoboypYozxA7oDi9IfYIwJBP4PG9hPZnCOvHX+sA2kDV+CYqUB6FC1A+HtwylVpBQDlg+g13e9SJEUZrSdwYjgERQt5PpCWYDtmhGBjZ/f+bi0se0PBmcrW86ARwdQumhpxv8ynlRJveOxYTFhxJ6OZVTTUdxf7H6Xr6WU8j5ZBncRSQYGAcuAXcBcEdlhjHnXGJM2ZfMjoDgQYYyJNsYsyuR0eSOtm6TZ6zdtrlqyKrPaz6JbzW50r9mdyI6RzqWKu5NSD0Gd52DTl/ZBaWaObrXLtmZzzfbifsX5S9BfiDkdw6LfMv9zRp+MZmrMVDo83OHGWH+lVIHnsZOYrrt82ibjqPM8PP/PnJ/PGUe2QFhreOr9295QrlsyHLbMhOF7bHdONqRKKi99+xJxl+JY/Pxi7vO776b9CUkJ/HHxH0lJTWFex3m37VdKeZ/cHAp5d9v4OSQnQvOh+XfNCg2hcgvY8LldffJWyVftBJ3az2Y7sAP4GB9Cm4RyNvEsn2+7vRtoQtQE4i7FMbbFWA3sSqmbeHZwT7wIv0yFWu1tSrz81Ox1uBgHOxbcvm/Pd3DlHNTPeRq9OgF16Fy9M7N2zWL/+f3Xt684tILIvZH0qdeHoHK5mClKKeUVPDu4b/7C9nu3GJb/167WBkrXgHX/sA9P04sOtxOeqrbOlUsNbjiYooWK8mHUh4gIp6+cZsy6MdT2r81r9XOwiqRSymt5bnBPSoT1n0KVx6GCG1quPj7QdBAcj4HfV9/YHn8S9n4Pj3bLtdmX/vf4MzBwIOuOrmPloZX8be3fSEhOYHyL8RTWvJpKqQx4bnDfNgviT7in1Z7m0RegWBmbii9NbARISq50yaT3Qs0XqFayGiPXjOTnIz/zRqM3eLhkLmUsUkp5Hc8M7inJsPYTKN/QttzdpfA9EDwA9i2HE4511KJn2XKVrZWrlyrkU4jQ4FASUxIJqRBC95rds36RUqrA8szgvnOBTTrRIhvL7+a2xn3scr7rP7VrzpzYnu2x7VkJfiCYme1mMvHxiTlLpq2U8noel4npegb50jWhZvusj89r9/pDYE/Y/CUkXQZfP7uCZB4JLBuYZ+dWSnkPz2u57/3eto6bD717ssU/9iqkJsOO+VCznXtS2CmlVDp3SXR0QeIFKB9oEyrfLQKq2glLkOsPUpVSKjs8r1vm0W42sN9tfc5PjIZSVaCaC1malFIqj3hecIe7L7ADlK4GT73n7lIopRTgid0ySimlsqTBXSmlvJAGd6WU8kIa3JVSygtpcFdKKS+kwV0ppbyQBnellPJCGtyVUsoLuS1BtjHmFHAwmy8vDZzOxeJ4moJc/4JcdyjY9de6Ww+JSJmsXuC24J4TxphNzmT/9lYFuf4Fue5QsOuvdXet7toto5RSXkiDu1JKeSFPDe5T3V0ANyvI9S/IdYeCXX+tuws8ss9dKaXUnXlqy10ppdQdeFxwN8Y8bYz51Rizzxgz0t3lyU/GmAPGmFhjTLQxZpO7y5PXjDEzjDEnjTHb023zN8YsN8bsdXwv5c4y5pVM6j7aGHPEcf+jjTHPuLOMecUYU9EY86MxZpcxZocxZohje0G595nV36X771HdMsYYX2AP0AaIA6KAHiKy060FyyfGmANAIxEpEGN9jTEtgXhgpojUdWybAJwVkfGON/dSIjLCneXMC5nUfTQQLyJ/d2fZ8pox5gHgARHZYoy5D9gMPAe8TMG495nVvxsu3H9Pa7kHA/tEZL+IXANmA53cXCaVR0RkNXD2ls2dgH85fv4X9h+918mk7gWCiBwTkS2Ony8Bu4AKFJx7n1n9XeJpwb0CcDjd73Fko9IeTIDvjTGbjTH93V0YNyknIsfA/icAyrq5PPltkDEmxtFt45XdEukZYyoDgcBGCuC9v6X+4ML997TgnlHyVM/pV8q5EBFpCLQDBjo+uquC459AVaABcAyY6N7i5C1jTHEgEhgqIhfdXZ78lkH9Xbr/nhbc44CK6X5/EDjqprLkOxE56vh+EpiP7aYqaE44+iTT+iZPurk8+UZETohIioikAmF48f03xhTGBravRORrx+YCc+8zqr+r99/TgnsUUN0YU8UY4wd0Bxa5uUz5whhTzPFwBWNMMeApYPudX+WVFgG9HD/3Aha6sSz5Ki2wOTyPl95/Y4wBpgO7RGRSul0F4t5nVn9X779HjZYBcAz/+RjwBWaIyAduLlK+MMY8jG2tAxQCZnl73Y0x4UAr7Ip4J4B3gAXAXKAScAjoKiJe9+Axk7q3wn4kF+AAMCCtD9qbGGOaA2uAWCDVsfl/sP3OBeHeZ1b/Hrhw/z0uuCullMqap3XLKKWUcoIGd6WU8kIa3JVSygtpcFdKKS+kwV0ppbyQBnellPJCGtyVUsoLaXBXSikv9B88wxSVOCojiAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.DataFrame(result.history)[['val_dense_{}_acc'.format(i) for i in range(1, 7)]].plot();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2363,  104,   12,    4,    0],\n",
       "       [ 183, 7605,  568,    0,    0],\n",
       "       [  20,   55, 2000,    6,    0],\n",
       "       [   0,    0,   83,   63,    0],\n",
       "       [   0,    0,    1,    1,    0]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_true=y_test[0], y_pred=np.argmax(n_digits, axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[   0,   14,    1,    1,    0,    0,    3,    0,    0,    0],\n",
       "       [   0, 3542,  100,    3,   23,    1,   13,   12,    0,    0],\n",
       "       [   0,  134, 2137,  222,   15,   20,   30,   97,    2,    0],\n",
       "       [   0,  112, 1029,  250,   33,  121,   39,   36,    5,    0],\n",
       "       [   0,  851,   91,    3,  239,    1,   20,   31,    0,    0],\n",
       "       [   0,   45,  481,  276,    2,  145,   65,   13,   25,    0],\n",
       "       [   0,  226,   89,   43,   70,   26,  354,   34,   11,    4],\n",
       "       [   0,  202,  335,   17,   21,    3,   18,  150,    1,    0],\n",
       "       [   0,   98,  126,   78,   21,   55,  189,   20,   34,    4],\n",
       "       [   0,   70,  216,   73,   12,   75,   64,   24,   22,    0]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_true=y_test[1], y_pred=np.argmax(digit1, axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    12704\n",
       "7      302\n",
       "2       55\n",
       "3        7\n",
       "dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(np.argmax(digit1, axis=1)).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    0.639425\n",
       "1    0.190006\n",
       "3    0.159244\n",
       "4    0.011172\n",
       "5    0.000153\n",
       "Name: 0, dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test[0].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "380px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
